Banning journal impact factors is bad for Dutch science

Recently, Utrecht University announced that it will ban journal impact factors from evaluation procedures of scientists. Such measurable performance indices are to be abandoned in favour of an “open science” system, which centralises the collective (team) at the expense of individual scientists.

However, we are concerned that Utrecht’s new “recognition and rewards” system will lead to randomness and a compromising of scientific quality, which will have serious consequences for the recognition and evaluation of Dutch scientists. In particular, it will have negative consequences for young scientists, who will no longer be able to compete internationally.  

Utrecht’s assertion that the journal impact factor plays a disproportionately large role in the evaluation of researchers is misleading. For a considerable number of research fields, impact factors are not that relevant. To account for field-specific cultures, the field-weighted citation impact score was developed, which compares the total citations actually received by a scientist with the average of the subject field. For example, research groups in medical imaging typically publish their results in technical journals with relatively low impact factors. Although not groundbreaking, however, the development of faster MRI methods is very important. The Dutch Research Council (NWO) takes this into account in its awarding policies. Accordingly, many personal grants have been awarded by the NWO’s career development programme to medical imaging researchers who never publish in high impact factor journals.

A second misconception is that a journal’s impact factor does not correlate with the quality of its publications. An average paper in a top journal, such as Nature, Science or Cell, requires much more work than an average paper in a technical journal. Top journals get assistance from world experts and thereby safeguard high impact and quality. This does not mean that every paper in Nature is by definition better than a publication in a technical journal, but, by and large, new technologies and concepts that overthrow dogmas are published in the top journals.

For the NWO’s “Veni, Vidi, Vici” talent programmes, the application format has changed radically over the past few years. The curriculum vitae with objective information on publications, citations, lectures and so on is replaced by a “narrative”. Reviewers will no longer grade the proposal and are forced instead to fill out lists with strengths and weaknesses, irrespective of their opinion of the proposal. For some NWO competitions, CVs are removed altogether because of the emphasis on “team science”.

The feedback from the assessment committees is disturbing. Members do not have a clue how to compare candidates, and googling their performance numbers is banned. Reviewers, often recruited from outside the Netherlands, complain about the time-consuming format and sometimes simply refuse to judge the narrative.

We believe that the NWO has the duty to allocate public funds in a way that supports the best and most talented scientists to discover new insights and innovations. We strongly support “recognition and rewards” for academics who are not exclusively science-oriented, but consider this the responsibility of the universities, not of the NWO. University HR policies must offer different career tracks for academics who excel in non-science competencies.

Quantitatively analysing a problem is an important feature of scientific practice, particularly in the medical, life and exact sciences. In these disciplines, creative solutions are sought for problems worldwide. Scientific success is therefore more easily measurable and comparable. For more qualitative sciences, it is understandable that other ways to assess success can be used. We strongly support diverse ways to evaluate different science disciplines and suggest that fields themselves determine how scientists in their discipline are assessed.

Utrecht’s policy puts a strong emphasis on open science, level of public engagement, public accessibility of data, composition of the research team and demonstrated leadership. These criteria are not scientific: they are political. Moreover, it is extremely difficult to measure them, let alone use them to conduct a fair comparison of different scientists. They should therefore not be the dominant criteria in the assessment of scientists. In particular, for the research track of the medical, life and exact sciences, internationally recognised and measurable criteria must be paramount.

The US, the world’s science powerhouse, is on a completely different trajectory from the Netherlands. Big public funders such as the National Institutes of Health (NIH) and the National Science Foundation (NSF) focus solely on scientific excellence and have not signed the Declaration on Research Assessment (Dora; also known as the San Francisco Declaration), which abolishes the impact factor as an evaluation index.

We believe that the NWO and the Dutch universities should maintain objective and measurable standards for academics who primarily focus on research. We prefer scientists who are optimised for generating the best science, not for writing the prettiest narrative. This will be the best way both to benefit society and to safeguard the Netherlands’ favourable position in international rankings.

Raymond Poot is an associate professor (UHD) at the Erasmus University Medical Center, Rotterdam. Willem Mulder is professor of precision medicine at the Radboud University Medical Center and the Eindhoven University of Technology. This is a translated and edited version of an article that was first published in the Dutch journal Science Guide, which was signed by another 172 academics.